On the State of the Art in Verification and Validation in Cyber Physical Systems

It is widely held that debugging cyber-physical systems (CPS) is challenging. However, few empirical studies quantitatively and qualitatively capture the state of the art and the state of the practice in debugging CPS and analyze what major research gaps remain. This paper presents an empirical study of verification and validation in CPS through three complementary methods: a structured on-line survey of CPS developers and researchers, semi-structured interviews with professional CPS developers from various backgrounds, and a qualitative analysis of state of the art in research related to CPS testing. We find that traditional verification and validation methodologies are not sufficient for cyber-physical systems, and we identify several potential avenues for future work. Our key findings include: (i) many CPS developers do not use traditional verification and validation methodologies and rely heavily on trial and error; (ii) simulation alone is not enough to capture dangerous bugs in CPS; (iii) it is widely acknowledged that the main challenges in CPS debugging are related to models of software systems, models of physics, and integration of cyber and physics models. These findings aid in identifying research directions to address the identified key challenges in CPS verification and validation.